System for entering handwritten data into computer generated forms

ABSTRACT

A method for entering data into a computer generated form including field areas of preselected height and width includes the steps of converting handwritten characters of arbitrary height which may be greater than the preselected height formed on the screen to computer generated characters and displaying the computer generated characters within a field area. Additionally, handwritten characters to be entered into several field areas are grouped, converted, and displayed in selected field areas.

This is a continuation of application Ser. No. 07/973,928 filed Nov. 10,1992, now abandoned, which is a continuation of application Ser. No.07/602,431 filed Oct. 22, 1990, now abandoned.

BACKGROUND OF THE INVENTION

The invention relates generally to computer data entry systems and moreparticularly to systems for entering handwritten data into a computer.

Systems that utilize a computer for displaying forms and entering datainto fields in the form are known. A typical computer generated form isdepicted in FIG. 1. In FIG. 1, a form 10 is displayed on a computerscreen 12. A pointer device 14 is used to make marks, such as editingcommands or handwritten alphanumeric characters, on the screen 10. Insome systems "electronic ink" is concurrently displayed on the screen 12as a mark is made by the pointer 14.

The form 10 includes field areas 16 and computer generated alphanumericcharacters forming text entries. Other features such as control buttons18 and lists 20 may also be included in a form. The field areas 16 andassociated text entries convey to a user that a certain type ofinformation is to be entered into a specific location of the form.

For example, the displayed text "Customer Name" and adjacent displayedfield area 16a convey to the user that the alphabetic charactersspelling out the customers name should be entered into the field 16a.Similarly, the customer knows that the numeric symbols for the number ofproducts is to be entered into the field area 16b.

FIG. 1A depicts a hand-held notebook style computer that can be utilizedin the field as a substitute for a clipboard. The form 10 is displayedon the screen 12 and the user fills in the form utilizing the pointer14. The details of the architecture and operation of this computer aredisclosed in a commonly-assigned patent application entitled HAND HELDNOTEBOOK STYLE COMPUTER, Ser. No. 365,147, which is hereby incorporatedby reference.

If the displayed form were printed on paper, the user would simply writethe required information in the correct field area utilizing a pen orpencil. If the letters did not quite fit in field areas there would beno effect on the utility of the form to convey or store the requiredinformation. Additionally, the user would not be required to fill in thefield areas in any particular order or time sequence.

Ideally, the computer data entry system would emulate the entry ofinformation on a paper form. However, existing systems have not yetachieved such ease of entry. The display of the form 10 on the screen 12and the recognition of handwritten characters and editing symbols isperformed by a computer program. One function of the computer program isto convert the handwritten information entered into a field area intodigital data stored in the computer memory for later use. Once thehandwritten characters are converted into digital form, the charactersare recognized by the computer program, and computer generatedcharacters corresponding to the handwritten characters are displayed inthe field and stored in the computer memory.

In addition to recognizing the handwritten characters the computerprogram must determine into which field the user intends to enterinformation. In some systems, the program displays informationindicating which field is ready to input data. For example, a field maybe displayed in reverse video. In existing systems, this conversion isinitiated when the user touches a special control button 18 with thepointer 14.

In another type of system the user must write characters that fit withinthe boundaries of the field. However, this system has several drawbacks.It is difficult to write and recognize small characters written on thescreen 12. Accordingly, the field areas 16 must be made large enough toaccommodate handwritten characters that are easily recognizable. Largefield areas, on the other hand, restrict the amount of information thatcan be entered into a displayed form 10 and decrease the efficiency ofthe system.

Another type of system displays the fields in a small format and thenpops-up a larger field for the entry of handwritten data. However, thissystem is slower because it requires extra steps and also allows entryinto only one field at a time.

Another known method for entering data into a computer generated form isto utilize pop-up keyboards. Generally, the keyboards pop-up in apredetermined sequence or in response to utilizing a control mechanism.The keys are used to enter computer generated characters into a fieldarea.

Accordingly, although the computerized entry of handwritten data intoforms is a great advance in user friendly computer interface technology,improvements are required to emulate the entry of data onto paper forms.

SUMMARY OF THE INVENTION

The present invention is a method for filling in a computer generatedform that is similar to the method for a filling in a paper form. Theuser may write handwritten characters of arbitrary height which may begreater than the height of the field areas of the form, fill in thefield areas in any particular order, or utilize a control button orother control mechanism to invoke conversion to computer generatedcharacters.

According to one aspect of the invention, the user writes handwrittencharacters on the screen and the computer writes electronic ink so thatthe characters are visible to the user. If the pen is held up for longerthan a predetermined time interval a time-out occurs and the handwrittencharacters are grouped according to the associated fields they arenearest to. Each group of handwritten characters are converted tocomputer generated characters and displayed within the associated fieldarea.

Other advantages and features of the invention will become apparent inview of the drawings and following detailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is schematic diagram of a computer generated form;

FIG. 1A is a perspective view of a hand-held notebook style computer;

FIG. 2A is schematic diagram of a computer generated form displayingelectronic ink of handwritten characters formed according to a preferredembodiment of the invention;

FIG. 2B is schematic diagram of a computer generated form displayingcomputer generated characters corresponding to the handwrittencharacters displayed in FIG. 2A;

FIG. 3 is a flow-chart illustrating a computer program for implementinga method for filling in a computer generated form;

FIGS. 4A and 4B are schematic diagrams of computer generated formsincluding pop-up keyboards;

FIG. 5 is a flow-chart illustrating a computer program for implementinga method of utilizing pop-up keyboards to fill in a computer generatedform; and

FIG. 6 is a block diagram of a digital system for executing the computerprograms for implementing the methods illustrated in FIGS. 3 and 5.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is a computer data entry system that emulates theease of entering handwritten information on a printed form. The methodof entering data will be described with reference to FIGS. 2A-2B.

In FIG. 2A the user has written the customer name, the quantity, and theprice over the appropriate field areas 16 of the form 10. There is norestriction on the order of entry, no limitations on the number offields areas 16 over which data can be entered, or any requirement thatthe handwritten characters be confined within the boundaries of thefield areas 16. Accordingly, the system emulates the ease of entry ofdata onto a paper form.

The conversion of the handwritten characters to computer generated"typed" characters is invoked by holding the pen up for a predeterminedtimeout period. The computer generated "typed" characters are displayedin the appropriate fields as depicted in FIG. 2B. Thus, the user in notrequired to utilize a control bottom or other control mechanism toinvoke conversion.

A computer program for implementing the method of filling in a displayedform described with reference to FIGS. 2A-2B will now be described withreference to the flowchart of FIG. 5.

Referring to FIG. 5, the program continually tests 110 whether the penhas come down to touch the screen. When the pen comes down it is firstdetermined 112 whether the pen is in a control area and, if so, theindicated control function is handled 114 and the program goes back totesting 110 whether the pen is down.

If the pen comes down and is not in a control area then the imagecurrently displayed on the screen is saved 116 in memory. The programthen repeatedly gets 118 the next point disposed under the moving pen,saves 120 the coordinates of the point in memory, and draws 120electronic ink on the display at the coordinates of the point until pencomes up. The program tests 124 whether the pen comes up after theelectronic ink is drawn 122 on the screen. The pen coming up indicatesthe end of a stroke.

Also, when the pen comes up the program determines whether the user hasfinished data entry and conversion of the handwritten character tocomputer generated "typed" characters is to be implemented.

This determination is implemented as follows: if it is determined 124that the pen is up the time the pen went up is noted. In practice, atimer may be triggered. The program the attempts to get 128 the nextpoint indicated by the pen. If the pen is down then the programimplements steps 120-124 to collect the points and display theelectronic ink for the next stroke. If the pen is still up then it isdetermined whether the timer indicates whether the pen has been up for apredetermined time period, e.g., greater than 0.5 sec. If not, theprogram keeps trying to get 128 the next point starting the next strokeuntil it is determined 132 that the pen has been up more than 0.5 sec.

If the pen is up more than 0.5 sec. then a time-out occurs and theprocess of converting the handwritten characters to computer generated,e.g., ASCII characters, is invoked. First, the saved screen image isrestored 134 by exchanging the screen image containing the electronicink indicating the coordinates of the strokes with the screen imagepreviously saved.

The strokes stored in the restored screen image are then formed intogroups associated with the field areas 16 of the form. First, thestrokes having coordinates disposed nearest a first field area areformed 136 into a first group. The first group of handwritten strokesare then converted 138 into ASCII characters and the ASCII charactersare placed 140 into the first field area. If it is determined 142 thatthere are still more strokes to process then a second group of strokesis formed 136, converted 138, and placed in a second field area. Thisprocess of conversion continues until it is determined 142 that thereare no more strokes to process. The program then returns to the step oftesting 110 whether the pen has come down on the screen.

In the example illustrated in FIGS. 2A and 2B, the user fills in thefield areas 16 of the form in any order. Note in FIG. 2A that thehandwritten characters need not be written inside the field areas.Accordingly, from the users point of view, utilizing the computergenerated field is equivalent to using a paper form.

When the user stops writing the computer detects time-out and thecharacters are grouped by field, converted to ASCII characters, anddisplayed in the appropriate fields as depicted in FIG. 2B.

An alternative system for entering information into a computer generatedform is depicted in FIGS. 4A and 4B. In FIG. 4A, an example of a pop-upkeyboard 60 for entering alphanumeric information into a field area 16is depicted. The pop-up keyboard 60 is displayed when a selected fieldarea 16s, in this example the "Customer Name" field area, is tappedtwice with the pointer. Accordingly, the present system allows the userfull flexibility to pop-up a keyboard for any field area withoututilizing a control button or other control mechanism.

The pop-up keyboard 60 includes a display area 62 and key areas 64 thatdisplay alphanumeric characters. The pop-up keyboard display 62originally displays the computer generated characters that weredisplayed in the selected field area 16s. The key areas 64 may be usedto insert computer generated characters at a location in the pop-upkeyboard display area 62 indicated by a cursor 66.

Additionally, handwritten data may be entered into the pop-up keyboarddisplay area in the same manner as described above for entry of datainto field areas 16. The pop-up keyboard is especially useful forentering symbols such as punctuation marks, symbols, and lower-caseletters that may not be recognizable by the handwriting recognitionalgorithm. The OK control button is touched to enter the data displayedin the pop-up keyboard display area 62s into the selected field area16s.

FIG. 4B depicts a numeric pop-up keyboard 60n that pops up in responseto tapping a selected field area into which numeric data is to beentered, e.g., field area 16n associated with the "Price" displayed textfield. The pop-up display 62n functions in a similar manner to thedisplay 62 in the alphanumeric keyboard 60.

During set up appropriate pop-up keyboards are associated with eachfield area. For example, the alphabetic keyboard depicted in FIG. 4Awould be associated with Customer Name field area and the calculatortype keyboard depicted in FIG. 4B would be associated with Price fieldarea.

A computer program for implementing the pop-up keyboard data entrymethod of the present invention is illustrated by the flowchart of FIG.5. Referring to FIG. 5, the program repeatedly tests 210-214 whether theuser indicates that a keyboard for a user-selected field is to bepopped-up. In the example, the user taps the selected field twice withthe pen.

Once a field is selected, the image displayed on the screen is saved 216and the data displayed in the selected field area is placed 218 in thefield area of the associated pop-up keyboard. The associated pop-upkeyboard with the placed data is then displayed 220.

The data displayed in the field area of the pop-up keyboard is thenmodified utilizing the key areas or by handwriting over the field areaas desired by the user. When the user wishes to enter the modified datainto the selected field area an exit control area, e.g., the OK button,is touched.

The saved screen area is then restored 226 by placing the modified datafrom the field area of the associated pop-up keyboard into the selectedfield area of the saved image.

The restored screen image is then displayed 228 with the modified datafrom the pop-up keyboard replacing the data previously displayed in theselected field area.

The editing functions that can be invoked in a preferred embodimentutilizing the pointer and handwritten editing symbols will now bebriefly described.

Text can be entered into any field in form by writing over the fieldarea. Electronic ink is displayed as the characters are written. Thehandwritten text is automatically recognized and replaced with "typed"computer generated characters and the electronic ink from the pointer iserased.

Characters are written adjacent to or overlapping existing text areappended to the right of existing text with no intervening space. Ifcharacters are written well to the right of existing text then a spaceis inserted before the new displayed characters.

The editing function of inserting characters between first and secondcharacters in an existing text field will now be briefly described. Aninsertion bar is invoked by touching the screen between the first andsecond characters. If the first stroke is a small quick dot in a fieldthen an insertion point is set at the stroke location. Characters to beentered may be written anywhere over the field except over the existingtext. When the pointer is held up longer that the predetermined intervalthe handwritten characters are recognized, computer generated charactercharacters are displayed after the first character, and text startingwith the second character is shifted to the right to make room for theinserted characters.

The editing function of deleting characters will now be brieflydescribed. To delete a single character a vertical line is drawn downthrough the character. The stroke starts above the character and goesdown through it. After the character is deleted the insertion bar isdisplayed to facilitate the editing function of replacing a characterwith another character. To delete more than one character a horizontalline is drawn through the characters to be deleted.

FIG. 6 is a block diagram of a digital system for implementing themethod of filling in forms described herein. In FIG. 6, a control unit320, e.g. a microprocessor is coupled to a video control system 322, amemory 324, and an overlay controller (OC) 326 by a system bus 330. Thememory stores firmware such as BIOS 324a and application programs 324b.

The invention has now been described with reference to a preferredembodiment. Modifications and substitutions will now be apparent topersons of ordinary skill in the art. In particular, variousconfigurations of a pop-up keyboard and different patterns forpopping-up the keyboard could be utilized. Accordingly, it is notintended to limit the scope of the invention except as provided by theappended claims.

What is claimed is:
 1. A method for entering data into a computer havinga dual function screen for displaying images stored in the computer andfor sensing screen coordinates selected by a pointer device, said methodcomprising the steps of:displaying, on the screen, a computer generatedimage including a displayed field area disposed at a predetermined,fixed location on the screen for the entry of at least two handwrittencharacters or symbols, the field area having a preselected and fixedheight and width; writing or drawing at least first and secondhandwritten characters or symbols by moving the pointer device insideand outside the field area while the pointer is touching the screen;maintaining the preselected height and width of the field area as thepointer device moves inside and outside the field area; determiningscreen coordinates of points, disposed inside and outside the field areaand located under the pointer, as the pointer moves inside and outsidethe field area; storing, as stroke data, the coordinates of pointsinside and outside the field area determined from where the pointertouches the screen; processing, in the computer, the stored stroke datato identify said first and second handwritten characters or symbols andto decode each identified handwritten character or symbol into acorresponding computer generated character or symbol; and displaying,under control of the computer, each computer generated character orsymbol within the field area so that each computer generated characteror symbol is disposed entirely within the field area.
 2. The methodaccording to claim 1 further comprising the steps of:detecting when thepointer touches the screen; detecting when the pointer stops touchingthe screen; measuring, in the computer, a time interval starting whenthe pointer stops touching the screen and resetting the time intervalwhen the pointer retouches the screen; and wherein the step ofprocessing the stored strokes to identify handwritten characters isperformed when the time interval exceeds a predetermined duration.
 3. Amethod for entering data into a computer having a dual function screenfor displaying images stored in the computer and for sensing screencoordinates selected by a pointer device, said method comprising thestep of:displaying, on the screen, a computer generated image includingfirst and second displayed field areas disposed at predeterminedlocations on the screen for the entry of at least two handwrittencharacters or symbols and each field area having a preselected and fixedheight and width; detecting when the pointer device touches the screen;detecting when the pointer device stops touching the screen; writing ordrawing with the pointer device a plurality of handwritten characters orsymbols by moving the pointer device inside and outside at least one ofthe first and second field areas while the pointer device is touchingthe screen; maintaining the preselected height and width of the firstand second field areas as the pointer device moves inside and outsidethe at least one of the first and second field areas; determining screencoordinates of points, disposed both inside and outside of the first andsecond field areas and located under the pointer device, as the pointerdevice is moved along the screen to write or draw said plurality ofhandwritten characters or symbols; storing, as stroke data, said screencoordinates; measuring, in the computer, a time interval starting whenthe pointer device stops touching the screen; resetting, in thecomputer, the time interval when the pointer device retouches thescreen; if the time interval exceeds a predetermined duration,processing, in the computer, the stored stroke data to identify saidplurality of handwritten characters and symbols and to decode eachidentified handwritten character or symbol into a corresponding computergenerated character or symbol; grouping, in the computer, those computergenerated characters or symbols decoded from stroke data disposednearest to the first field area into a first group; displaying, undercontrol of the computer, the first group of decoded computer generatedcharacters or symbols within the first field area so that each computergenerated character or symbol in the first group is disposed entirelywithin the first field; grouping, in the computer, those computergenerated characters or symbols decoded from stroke data disposednearest to the second field area into a second group; and displaying,under control of the computer, the second group of decoded computergenerated characters or symbols within the second field area so thateach computer generated character or symbol in the second group isdisposed entirely within the second field.
 4. A method for entering datainto a computer having a dual function screen for displaying imagesstored in the computer and for sensing screen coordinates selected by apointer device, said method comprising the steps of:displaying, on thescreen, a computer generated image including first and second fieldareas disposed at predetermined locations on the screen for the entryand display of information, said first field area for entering at leasttwo handwritten characters and said first field area having apreselected and fixed first height and first width defining a firstfield boundary, said second field area for entering at least twohandwritten characters and said second field area having a preselectedand fixed second height and second width defining a second fieldboundary; writing or drawing with the pointer device a first set ofhandwritten characters within or near said first field area, said firstset of handwritten characters including at least two handwrittencharacters and where at least a portion of one of said two handwrittencharacters extends beyond said first field boundary; digitizing signalsrepresentative of said first set of handwritten characters; grouping, inthe computer, said digitized signals of said first set of handwrittencharacters into a first group of digitized signals, wherein some of saiddigitized signals grouped in said first group correspond to said portionof a handwritten character extending beyond said first field boundary;and processing thereafter, in the computer, the first group of digitizedsignals to identify the first set of handwritten characters as a firstgroup of computer generated characters.
 5. The method set forth in claim4 further comprising the step of:displaying, under control of thecomputer, said first group of computer generated characters within saidfirst field area.
 6. The method set forth in claim 4 further comprisingthe step of:measuring, in the computer, a time interval starting whenthe pointer device stops touching the screen; and wherein saidprocessing step begins if the time interval exceeds a predeterminedduration.
 7. The method set forth in claim 4 further comprising thesteps of:writing or drawing with the pointer device a second set ofhandwritten characters within or near said second field area, saidsecond set of handwritten characters including at least two handwrittencharacters and where at least a portion of one of said two handwrittencharacters extends beyond said second field boundary; digitizing signalsrepresentative of said second set of handwritten characters; grouping,in the computer, said digitized signals of said second set ofhandwritten characters into a second group of digitized signals whereinsome of said digitized signals grouped in said second group correspondto said portion of a handwriting character extending beyond said secondfield boundary; processing thereafter, in the computer, the second groupof digitized signals to identify the second set of handwrittencharacters as computer generated characters.
 8. The method set forth inclaim 7 further comprising the step of:displaying, under control of thecomputer, said second group of computer generated characters within saidsecond field area.
 9. The method set forth in claim 7 further comprisingthe step of:measuring, in the computer, a time interval starting whenthe pointer device stops touching the screen; and wherein saidprocessing steps begin if the time interval exceeds a predeterminedduration.
 10. A method for entering data into a computer having a dualfunction screen for displaying images stored in the computer and forsensing screen coordinates selected by a pointer device, said methodcomprising the steps of:displaying, on the screen, a computer generatedimage including first and second field areas disposed at predeterminedlocations on the screen for the entry and display of information, saidfirst field area for entry of at least two handwritten symbols and saidfirst field area having a preselected and fixed first height and firstwidth, said second field area for entry of at least two handwrittensymbols and said second field area having a preselected and fixed secondheight and second width; using the pointer device to write or draw aplurality of handwritten symbols in each field area wherein at least aportion of the symbols in the first field area extends beyond the firstfield area and not into the second field area; digitizing signalsrepresentative of said plurality of handwritten symbols; grouping, inthe computer, said digitized signals positioned within and near thefirst field area into a first group of digitized signals where at leasta portion of said digitized signals corresponds to the portion of saidplurality of handwritten symbols outside the first field area; andprocessing thereafter, in the computer, the first group of digitizedsignals to identify at least two handwritten symbols as a first group ofcomputer generated symbols characters.
 11. A method for entering datainto a computer having a dual function screen for displaying imagesstored in the computer and for sensing screen coordinates selected by apointer device, said method comprising the steps of:displaying, on thescreen, a computer generated image including first and second fieldareas disposed at predetermined locations on the screen for the entryand display of information, said first field area for the entry of atleast two handwritten characters and said first field area having afirst horizontal boundary having a fixed vertical position on thescreen, said second field area for the entry of at least two handwrittencharacters and said second field area having a second horizontalboundary having a second fixed vertical position on the screen; writingor drawing with the pointer device a first set of handwritten characterswithin or near said first field area, said first set of handwrittencharacters including at least two handwritten characters and where atleast a portion of one of said two handwritten characters extends acrosssaid first horizontal boundary; digitizing signals representative ofsaid first set of handwritten characters; grouping, in the computer,said digitized signals of said first set of handwritten characters intoa first group of digitized signals, wherein at least a portion of saiddigitized signals grouped in said first group correspond to said portionof a handwritten character extending across said first horizontalboundary; and processing thereafter, in the computer, the first group ofdigitized signals to identify the first set of handwritten characters asa first group of computer generated characters.
 12. The method set forthin claim 11 further comprising the step of:displaying, under control ofthe computer, said first group of computer generated characters withinsaid first field area.
 13. The method set forth in claim 11 furthercomprising the step of:measuring, in the computer, a time intervalstarting when the pointer device stops touching the screen; and whereinsaid processing step begins if the time interval exceeds a predeterminedduration.
 14. The method set forth in claim 11 further comprising thesteps of:writing or drawing with the pointer device a second set ofhandwritten characters within or near said second field area, saidsecond set of handwritten characters including at least two handwrittencharacters; digitizing signals representative of said second set ofhandwritten characters; grouping, in the computer, said digitizedsignals of said second set of handwritten characters into a second groupof digitized signals; processing thereafter, in the computer, the secondgroup of digitized signals to identify the second set of handwrittencharacters as computer generated characters.
 15. The method set forth inclaim 14 further comprising the step of:displaying, under control of thecomputer, said second group of computer generated characters within saidsecond field area.
 16. The method set forth in claim 14 furthercomprising the step of:measuring, in the computer, a time intervalstarting when the pointer device stops touching the screen; and whereinsaid processing steps begin if the time interval exceeds a predeterminedduration.
 17. The method according to claim 11 wherein the horizontalboundary comprises a lower horizontal boundary.
 18. The method accordingto claim 11 wherein the horizontal boundary comprises an upperhorizontal boundary.
 19. The method according to claim 11 furthercomprising the step of:measuring, in the computer, a time intervalstarting when the pointer stops touching the screen and ending when thepointer retouches the screen; and wherein the step of processing thestored strokes to identify handwritten characters is performed when thetime interval exceeds a predetermined duration.
 20. A method forentering data into a computer having a dual function screen fordisplaying images stored in the computer and for sensing screencoordinates selected by a pointer device, said method comprising thesteps of:displaying, on the screen, a computer generated image includingfirst and second field areas disposed at predetermined locations on thescreen for the entry and display of characters, said first field areafor entering at least two handwritten characters and said first fieldarea having a preselected and fixed first height and first width, saidsecond field area for entering at least two handwritten characters andsaid second field area having a preselected and fixed second height andsecond width; writing or drawing with the pointer device a first set ofat least two handwritten characters, wherein at least one of said firstset of handwritten characters is at least partially outside said firstfield area; writing or drawing with the pointer device a second set ofat least two handwritten characters, wherein at least one of said secondset of handwritten characters is at least partially outside said secondfield area; digitizing signals representative of said first and secondsets of handwritten characters irrespective of whether portions of saidfirst or second sets of handwritten characters are within said first orsecond field areas; grouping, in the computer, said digitized signalscorresponding to said first set of handwritten characters into a firstgroup of digitized signals; grouping, in the computer, said digitizedsignals corresponding to said second set of handwritten characters intoa second group of digitized signals; thereafter, processing, in thecomputer, the first group of digitized signals to identify the first setof handwritten characters as a first group of computer generatedcharacters and processing, in the computer, the second group ofdigitized signals to identify the second set of handwritten charactersas a second group of computer generated characters; displaying, in saidfirst field area, said first group of computer generated characters; anddisplaying, in said second field area, said second group of computergenerated characters.